Apparatus system and method for human-machine-interface

ABSTRACT

There is provided a 3D human machine interface (“3D HMI”), which 3D HMI may include: (1) an image acquisition assembly, (2) an initializing module, (3) an image segmentation module, (4) a segmented data processing module, (5) a scoring module, (6) a projection module, (7) a fitting module, (8) a scoring and error detection module, (9) a recovery module, (10) a three dimensional correlation module, (11) a three dimensional skeleton prediction module, (12) an output module and (13) a depth extraction module.

RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 12/517,210 filed Jun. 2, 2009, which application isa 371 National Stage Application of PCT Application PCT/IL2007/000475filed Apr. 15, 2007. U.S. patent application Ser. No. 12/517,210 is acontinuation-in-part of U.S. patent application Ser. No. 11/572,958filed Jan. 30, 2007, U.S. Pat. No. 8,114,172, which is a 371 NationalStage Application of PCT Application PCT/IL2005/000813 filed Jul. 31,2005, which claim the benefit of U.S. 60/592,136, filed Jul. 30, 2004.U.S. patent application Ser. No. 12/517,210 is a continuation-in-part ofPCT Application PCT/IL2006/001254 filed Oct. 31, 2006, which claims thebenefit of U.S. 60/731,274, filed Oct. 31, 2005.

Each of the aforementioned applications is hereby incorporated byreference in its entirety.

FIELD OF THE INVENTION

The present invention generally relates to user interfaces and moreparticularly to methods and systems of 3D Human-Machine-Interface.

BACKGROUND OF THE INVENTION

One of the largest patterns in the history of software is the shift fromcomputation-intensive design to presentation-intensive design. Asmachines have become more and more powerful, inventors have spent asteadily increasing fraction of that power on presentation. The historyof that progression can be conveniently broken into three eras: batch(1945-1968), command-line (1969-1983) and graphical (1984 and after).The story begins, of course, with the invention of the digital computer.The opening dates on the latter two eras are the years when vital newinterface technologies broke out of the laboratory and began totransform users' expectations about interfaces in a serious way. Thosetechnologies were interactive timesharing and the graphical userinterface.

In the batch era, computing power was extremely scarce and expensive.The largest computers of that time commanded fewer logic cycles persecond than a typical toaster or microwave oven does today, and quite abit fewer than today's cars, digital watches, or cellphones. Userinterfaces were, accordingly, rudimentary. Users had to accommodatecomputers rather than the other way around; user interfaces wereconsidered overhead, and software was designed to keep the processor atmaximum utilization with as little overhead as possible.

The input side of the user interfaces for batch machines were mainlypunched cards or equivalent media like paper tape. The output side addedline printers to these media. With the limited exception of the systemoperator's console, human beings did not interact with batch machines inreal time at all.

Submitting a job to a batch machine involved, first, preparing a deck ofpunched cards describing a program and a dataset. Punching the programcards wasn't done on the computer itself, but on specializedtypewriter-like machines that were notoriously balky, unforgiving, andprone to mechanical failure. The software interface was similarlyunforgiving, with very strict syntaxes meant to be parsed by thesmallest possible compilers and interpreters.

Once the cards were punched, one would drop them in a job queue andwait. Eventually, operators would feed the deck to the computer, perhapsmounting magnetic tapes to supply another dataset or helper software.The job would generate a printout, containing final results or (all toooften) an abort notice with an attached error log. Successful runs mightalso write a result on magnetic tape or generate some data cards to beused in later computation.

The turnaround time for a single job often spanned entire days. If onewere very lucky, it might be hours; real-time response was unheard of.But there were worse fates than the card queue; some computers actuallyrequired an even more tedious and error-prone process of toggling inprograms in binary code using console switches. The very earliestmachines actually had to be partly rewired to incorporate program logicinto themselves, using devices known as plug boards.

Early batch systems gave the currently running job the entire computer;program decks and tapes had to include what we would now think of asoperating-system code to talk to I/O devices and do whatever otherhousekeeping was needed. Midway through the batch period, after 1957,various groups began to experiment with so-called “load-and-go” systems.These used a monitor program which was always resident on the computer.Programs could call the monitor for services. Another function of themonitor was to do better error checking on submitted jobs, catchingerrors earlier and more intelligently and generating more usefulfeedback to the users. Thus, monitors represented a first step towardsboth operating systems and explicitly designed user interfaces.

Command-line interfaces (CLIs) evolved from batch monitors connected tothe system console. Their interaction model was a series ofrequest-response transactions, with requests expressed as textualcommands in a specialized vocabulary. Latency was far lower than forbatch systems, dropping from days or hours to seconds. Accordingly,command-line systems allowed the user to change his or her mind aboutlater stages of the transaction in response to real-time ornear-real-time feedback on earlier results. Software could beexploratory and interactive in ways not possible before. But theseinterfaces still placed a relatively heavy mnemonic load on the user,requiring a serious investment of effort and learning time to master.

Command-line interfaces were closely associated with the rise oftimesharing computers. The concept of timesharing dates back to the1950s; the most influential early experiment was the MULTICS operatingsystem after 1965; and by far the most influential of present-daycommand-line interfaces is that of Unix itself, which dates from 1969and has exerted a shaping influence on most of what came after it.

The earliest command-line systems combined teletypes with computers,adapting a mature technology that had proven effective for mediating thetransfer of information over wires between human beings. Teletypes hadoriginally been invented as devices for automatic telegraph transmissionand reception; they had a history going back to 1902 and had alreadybecome well-established in newsrooms and elsewhere by 1920. In reusingthem, economy was certainly a consideration, but psychology and the Ruleof Least Surprise mattered as well; teletypes provided a point ofinterface with the system that was familiar to many engineers and users.

The widespread adoption of video-display terminals (VDTs) in themid-1970s ushered in the second phase of command-line systems. These cutlatency further, because characters could be thrown on the phosphor dotsof a screen more quickly than a printer head or carriage can move. Theyhelped quell conservative resistance to interactive programming bycutting ink and paper consumables out of the cost picture, and were tothe first TV generation of the late 1950s and 60s even more iconic andcomfortable than teletypes had been to the computer pioneers of the1940s.

Just as importantly, the existence of an accessible screen, atwo-dimensional display of text that could be rapidly and reversiblymodified made it economical for software designers to deploy interfacesthat could be described as visual rather than textual. The pioneeringapplications of this kind were computer games and text editors; closedescendants of some of the earliest specimens, such as rogue(6), andvi(1), are still a live part of Unix tradition.

Screen video displays were not entirely novel, having appeared onminicomputers as early as the PDP-1 back in 1961. But until the move toVDTs attached via serial cables, each exceedingly expensive computercould support only one addressable display, on its console. Under thoseconditions it was difficult for any tradition of visual UI to develop;such interfaces were one-offs built only in the rare circumstances whereentire computers could be at least temporarily devoted to serving asingle user.

There were sporadic experiments with what we would now call a graphicaluser interface as far back as 1962 and the pioneering SPACEWAR game onthe PDP-1. The display on that machine was not just a characterterminal, but a modified oscilloscope that could be made to supportvector graphics. The SPACEWAR interface, though mainly using toggleswitches, also featured the first crude trackballs, custom-built by theplayers themselves. Ten years later, in the early 1970s theseexperiments spawned the video-game industry, which actually began withan attempt to produce an arcade version of SPACEWAR.

The PDP-1 console display had been descended from the radar displaytubes of World War II, twenty years earlier, reflecting the fact thatsome key pioneers of minicomputing at MIT's Lincoln Labs were formerradar technicians. Across the continent in that same year of 1962,another former radar technician was beginning to blaze a different trailat Stanford Research Institute. His name was Doug Engelbart. He had beeninspired by both his personal experiences with these very earlygraphical displays and by Vannevar Bush's seminal essay As We May Think,which had presented in 1945 a vision of what we would today callhypertext.

In December 1968, Engelbart and his team from SRI gave a 90-minutepublic demonstration of the first hypertext system, NLS/Augment. Thedemonstration included the debut of the three-button mouse (Engelbart'sinvention), graphical displays with a multiple-window interface,hyperlinks, and on-screen video conferencing. This demo was a sensationwith consequences that would reverberate through computer science for aquarter century, up to and including the invention of the World Wide Webin 1991.

So, as early as the 1960s it was already well understood that graphicalpresentation could make for a compelling user experience. Pointingdevices equivalent to the mouse had already been invented, and manymainframes of the later 1960s had display capabilities comparable tothose of the PDP-1. A very early video game in 1968, on the console of aUnivac 1108 mainframe would cost nearly forty-five million dollars ifyou could buy it in 2004.

Video games became mass-market devices earlier than computers becausethey ran hardwired programs on extremely cheap and simple processors.But on general-purpose computers, oscilloscope displays became anevolutionary dead end. The concept of using graphical, visual interfacesfor normal interaction with a computer had to wait a few years and wasactually ushered in by advanced graphics-capable versions of theserial-line character VDT in the late 1970s.

Since the earliest PARC systems in the 1970s, the design of GUIs hasbeen almost completely dominated by what has come to be called the WIMP(Windows, Icons, Mice, Pointer) model pioneered by the Alto. Consideringthe immense changes is in computing and display hardware over theensuing decades, it has proven surprisingly difficult to think beyondthe WIMP.

A few attempts have been made. Perhaps the boldest is in VR (virtualreality) interfaces, in which users move around and gesture withinimmersive graphical 3-D environments. VR has attracted a large researchcommunity since the mid-1980s. A fundamental problem, familiar for manyyears to designers of flight simulators, is the way VR can confuse thehuman proprioceptive system; VR motion at even moderate speeds caninduce dizziness and nausea as the brain tries to reconcile the visualsimulation of motion with the inner ear's report of the body'sreal-world motions.

Jef Raskin's THE project (The Humane Environment) is exploring the zoomworld model of GUIs, distinct in that it spatializes them without going3D. In THE, the screen becomes a window on a 2-D virtual world wheredata and programs are organized by spatial locality. Objects in theworld can be presented at several levels of detail depending on one'sheight above the reference plane, and the most basic selection operationis to zoom in and land on them.

The Lifestreams project at Yale University goes in a completely oppositedirection, actually de-spatializing the GUI. The user's documents arepresented as a kind of world-line or temporal stream which is organizedby modification date and can be filtered in various ways.

All three of these approaches discard conventional filesystems in favorof a context that tries to avoid naming things and using names as themain form of reference. This makes them difficult to match with thefilesystems and hierarchical namespaces of Unix's architecture, whichseems to be one of its most enduring and effective features.Nevertheless, it is possible that one of these early experiments may yetprove as seminal as Engelbart's 1968 demo of NLS/Augment.

There is a need in the field of user interfaces for an improved systemand method of a Human-Machine-Interface.

SUMMARY OF THE INVENTION

According to some embodiments of the present invention, there isprovided a 3D human machine interface (“3D HMI”), which 3D HMI mayinclude (1) an image acquisition assembly, (2) an initializing module,(3) an image segmentation module, (4) a segmented data processingmodule, (5) a scoring module, (6) a projection module, (7) a fittingmodule, (8) a scoring and error detection module, (9) a recovery module,(10) a three dimensional correlation module, (11) a three dimensionalskeleton prediction module, and (12) an output module.

According to some embodiments of the present invention, the imageacquisition assembly may be adapted to acquire a set of images, whereinsubstantially each image is associated with a different point in time.According to some further embodiments of the present invention, theimages may be of a single user or multiple users.

According to some embodiments of the present invention, theinitialization module may be adapted to detect and define the user's (1)colors, (2) organ parameters, (3) surroundings, and (4) other parameterswhich are associated with the user.

According to some embodiments of the present invention, the user may beany person and/or animal and/or moving object which enters the frame.

According to some embodiments of the present invention, the imagesegmentation module may be adapted to extract segmented data from theimage. According to yet further embodiments of the present invention,the segmented data may also comprise:

-   -   Color    -   Movement    -   Edge detection    -   Texture.

According to some embodiments of the present invention, the segmenteddata processing module may be adapted to process the segmented data.According to yet further embodiments of the present invention, thesegmented data may be processed in the following way:

-   -   Color—using known color parameters to detect elements and/or        light changes, for example, using skin color to detect palms and        face.    -   Movement—detecting moving elements in the frame.    -   Background removal.    -   Edge detection—detect the edges of the image.    -   Texture—using known texture parameters to detect elements.

According to some embodiments of the present invention, the segmenteddata processing module may be adapted to detect deviation in thedistance of an organ from the image acquisition assembly, in accordancewith the deviation of the organs relative size.

According to some embodiments of the present invention, the scoringmodule may be adapted to (1) examine the processed segmented data, (2)estimate the quality of the processed segmented data, and according tothe quality (3) decide which portions of the segmented data are reliableenough to be used by the HMI system.

According to some embodiments of the present invention, the threedimensional skeleton prediction module may be adapted to predict theposition of the three dimensional skeleton which will have the bestmatch or correlation with the processed image.

According to further embodiments of the present invention, the threedimensional prediction module may use constraints which derive from thetype of skeleton used, for example, if the skeleton is of a humanfigure, the head of the skeleton can't rotate 360 degrees.

According to yet further embodiments of the present invention, the threedimensional prediction module may also use a set of dynamic and motionprocesses to predict the position of the three dimensional skeleton.

According to some embodiments of the present invention, the projectionmodule may be adapted to project the skeleton onto the image. Accordingto some further embodiments of the present invention, the projection maybe applied in a two-dimensional plane.

According to some embodiments of the present invention, the fittingmodule may be adapted to fit segmented data to the projected skeleton.According to some further embodiments of the present invention, thefitting module may be adapted to associate portions of the segmenteddata with portions of the projected skeleton.

According to some embodiments of the present invention, the scoring anderror detection module may be adapted: (1) to examine the processedskeleton after it was associated with segmented data, (2) to evaluatethe fitting quality of said skeleton and (3) determine whether an errorhas occurred during the skeleton prediction process or the associationof segmented data.

According to some embodiments of the present invention, the recoverymodule may be adapted to recover from a detected error. According tosome further embodiments of the present invention, the recovery may be aprocess of multiple processing layers, resegmenting the image, using the3D skeleton motion history to repredict the correct position,reprojecting and refitting the 3D skeleton. The recovery module may alsodecide to skip a frame if the image information is corrupt.

According to some embodiments of the present invention, the threedimensional correlation module may be adapted to update the position ofthe three dimensional skeleton in accordance with the position of thefitted skeleton.

According to some further embodiments of the present invention, saidupdating process associates the 3D skeleton on the fitted skeleton, fitsbetween the 3D skeleton and the fitted skeleton, and updates the 3Dskeleton to the correct position.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features, and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanying drawings in which:

FIG. 1, there is shown a block diagram depicting a system in accordancewith some embodiments of the present invention.

FIG. 2, there is shown a flow-chart depicting the steps of an HMI systemin accordance with some embodiments of the present invention.

FIG. 3, there is shown a block diagram depicting a system in accordancewith some embodiments of the present invention.

FIG. 4, there is shown a flow-chart depicting the steps of an HMI systemin accordance with some embodiments of the present invention.

FIG. 5, there is shown a block diagram depicting a system in accordancewith some embodiments of the present invention.

FIG. 6, there is shown a flow-chart depicting the steps of an HMI systemin accordance with some embodiments of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the invention.However, it will be understood by those skilled in the art that thepresent invention may be practiced without these specific details. Inother instances, well-known methods, procedures, components and circuitshave not been described in detail so as not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “computing”,“calculating”, “determining”, or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulate and/or transform data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

Embodiments of the present invention may include apparatuses forperforming the operations herein. This apparatus may be speciallyconstructed for the desired purposes, or it may comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Such a computer program may be stored ina computer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs,magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs) electrically programmable read-only memories (EPROMs),electrically erasable and programmable read only memories (EEPROMs),magnetic or optical cards, or any other type of media suitable forstoring electronic instructions, and capable of being coupled to acomputer system bus.

According to some embodiments of the present invention, there isprovided a 3D human machine interface (“3D HMI”), which 3D HMI mayinclude: (1) an image acquisition assembly, (2) an initializing module,(3) an image segmentation module, (4) a segmented data processingmodule, (5) a scoring module, (6) a projection module, (7) a fittingmodule, (8) a scoring and error detection module, (9) a recovery module,(10) a three dimensional correlation module, (11) a three dimensionalskeleton prediction module, and (12) an output module.

According to some embodiments of the present invention, the imageacquisition assembly may be adapted to acquire a set of images, whereinsubstantially each image is associated with a different point in time.According to some further embodiments of the present invention, theimages may be of a single user or multiple users.

According to some embodiments of the present invention, theinitialization module may be adapted to detect and define the user's:(1) colors, (2) organ parameters, (3) surroundings, and (4) otherparameters which are associated with the user, and decide on the bestway for image segmentation in the next steps (thresholds, score forevery image segmentation, etc.)

According to some embodiments of the present invention, the imagesegmentation module may be adapted to extract segmented data from theimage. According to yet further embodiments of the present invention,the segmented data may also comprise:

-   -   Color    -   Movement    -   Edge detection    -   Texture.

According to some embodiments of the present invention, the segmenteddata processing module may be adapted to process the segmented data.According to yet further embodiments of the present invention, thesegmented data may be processed in the following way:

-   -   Color—using known color parameters to detect elements and/or        light changes, for example, using skin color to detect palms and        face.    -   Movement—detecting moving elements in the frame.    -   Background removal.    -   Edge detection—detect the edges of the image.    -   Texture—using known texture parameters to detect elements.

According to some embodiments of the present invention, the scoringmodule may be adapted to: (1) examine the processed segmented data, (2)estimate the quality of the processed segmented data, and according tothe quality (3) decide which portions of the segmented data are reliableenough to be used by the HMI system.

According to some embodiments of the present invention, the threedimensional skeleton prediction module may be adapted to predict theposition of the three dimensional skeleton which will have the bestmatch or correlation with the processed image.

According to further embodiments of the present invention, the threedimensional prediction module may use constraints which derive from thetype of skeleton used, for example, if the skeleton is of a humanfigure, the head of the skeleton can't rotate 360 degrees.

According to yet further embodiments of the present invention, the threedimensional prediction module may use a set of dynamic and motionprocesses to predict the position of the three dimensional skeleton.

According to some embodiments of the present invention, the projectionmodule may be adapted to project the skeleton onto the image. Accordingto some further embodiments of the present invention, the projection maybe applied in the two-dimensional plane.

According to some embodiments of the present invention, the fittingmodule may be adapted to fit segmented data to the projected skeleton.According to some further embodiments of the present invention, thefitting module may be adapted to associate portions of the segmenteddata with portions of the projected skeleton.

According to some embodiments of the present invention, the scoring anderror detection module may be adapted: (1) to examine the processedskeleton after it was associated with segmented data, (2) to evaluatethe fitting quality of said skeleton and (3) determine whether an errorhas occurred during the skeleton prediction process or the associationof segmented data.

According to some embodiments of the present invention, the recoverymodule may be adapted to recover from a detected error. According tosome further embodiments of the present invention, the recovery may be aprocess of multiple processing layers, resegmenting the image, using the3D skeleton motion history to repredict the correct position,reprojecting and refitting the 3D skeleton. The recovery module may alsodecide to skip a frame if the image information is corrupt.

According to some embodiments of the present invention, the threedimensional correlation module may be adapted to update the position ofthe three dimensional skeleton in accordance with the position of thefitted skeleton.

According to some further embodiments of the present invention, saidupdating process associates the 3D skeleton on the fitted skeleton, fitsbetween the 3D skeleton and the fitted skeleton, and updates the 3Dskeleton to the correct position.

Turning now to FIG. 1, there is shown an exemplary HMI system inaccordance with some embodiments of the present invention, which systemmay be best described in conjunction with FIG. 2, there is shown a flowchart depicting the steps of such an HMI system.

According to some embodiments of the present invention, FIG. 1 shows a3D human machine interface (“3D HMI”), which 3D HMI may include: (1) animage acquisition assembly 1000, (2) an initializing module 1100, (3) animage segmentation module 1200, (4) a segmented data processing module1300, (5) a scoring module 1400, (6) a projection module 1450, (7) afitting module 1500, (8) a scoring and error detection module 1550, (9)a recovery module 1600, (10) a three dimensional correlation module1700, (11) a three dimensional skeleton prediction module 1800, and (12)an output module 1900.

According to some embodiments of the present invention, the imageacquisition assembly may be adapted to acquire a set of images, as seenin step 2000, wherein substantially each image is associated with adifferent point in time. According to some further embodiments of thepresent invention, the images may be of a single user or multiple users.

According to yet further embodiments of the present invention the imageacquisition assembly may comprise of a digital camera, a web camera, afilm camera, a video camera, a web camera, a digital video camera, ananalogue video camera, a stereo-camera and/or any other camera knowntoday or to be devised in the future.

According to some embodiments of the present invention, after the systemhas acquired one or more images, the system may enter an initializationphase, step 2100, which is performed by the initialization module 1100.The initialization module may be adapted to detect and define theuser's: (1) colors, (2) organ parameters, (3) surroundings, and (4)other parameters which are associated with the user.

According to some embodiments of the present invention, the system maybe adapted to extract segmentation data, as shown in step 2200, whichsegmented data may comprise:

-   -   Color    -   Movement    -   Edge detection    -   Texture.

According to yet further embodiments of the present invention, the imagesegmentation module 1200 may be adapted to extract the segmented datafrom the image.

According to some embodiments of the present invention, the system maybe adapted to process the segmented data, as shown in step 2300.According to yet further embodiments of the present invention, thesegmented data may be processed in the following way:

-   -   Color—using known color parameters to detect elements and/or        light changes, for example, using skin color to detect palms and        face.    -   Movement—detecting moving elements in the frame.    -   Background removal.    -   Edge detection—detect the edges in the image.    -   Texture—using known texture parameters to detect elements.

According to yet further embodiments of the present invention, thesegmented data processing module, 1300, may be adapted to process thesegmented data.

According to some embodiments of the present invention, the system maybe adapted to evaluate the quality of the segmented data, as shown instep 2400, the evaluation is performed by: (1) examining the processedsegmented data, (2) estimating the quality of the processed segmenteddata, and according to the estimated quality (3) deciding which portionsof the segmented data are reliable enough to be used by the HMI system.

According to yet further embodiments of the present invention, thescoring module 1400, may be adapted evaluate the quality of thesegmented information.

According to further embodiments of the present invention, the systemmay be adapted to predict the position of the three dimensionalskeleton, as shown in step 2800, which position will have the best matchor correlation with the processed image. According to some furtherembodiments of the present invention, the prediction may be moreaccurate with the use of constraints which derive from the type ofskeleton used, for example, if the skeleton is of a human figure, thehead of the skeleton can't rotate 360 degrees without a motion of theshoulders.

According to some embodiments of the present invention, the predictionsequence may also use a set of dynamic and motion processes and so on.

According to some embodiments of the present invention, the threedimensional skeleton prediction module 1800 may be adapted to predictthe position of the three dimensional skeleton.

According to some embodiments of the present invention, the system maybe further adapted to project the skeleton onto the image, as shown instep 2450. According to some further embodiments of the presentinvention, the projection may be applied in the two-dimensional plane.

According to some embodiments of the present invention, the projectionmodule, 2450, may be adapted to project the skeleton onto the image.

According to some embodiments of the present invention, the system maybe further adapted to fit the segmented data with the projectedskeleton, as shown in step 2500. According to some further embodimentsof the present invention, the fitting process may comprise theassociation of portions of the segmented data with portions of theprojected skeleton.

According to some embodiments of the present invention, fitting thesegmented data may comprise associating portions of the extractedsegmented data with current skeleton parameters, which current skeletonparameters may support the associated portions of extracted segmenteddata.

According to some further embodiments of the present invention, theoutcome of this process is a “fitted skeleton”.

According to some further embodiments of the present invention, thefitting module, 2500, may be adapted to associate the segmented datawith the projected skeleton.

According to some embodiments of the present invention, the system maybe further adapted to give a score to the fitted skeleton and detecterrors, as shown in step 2550. According to some embodiments of thepresent invention, giving a score and detecting errors may comprise: (1)examining the fitted skeleton, (2) evaluating the fitting quality ofsaid skeleton and (3) determining whether an error has occurred duringthe skeleton prediction process or the association of segmented data.

According to some embodiments of the present invention, the scoring anderror detection module 1550, may be adapted to give scores and detecterrors.

According to some embodiments of the present invention, if an error wasdetected during step 2550, the system may enter a recovery phase, asshown in step 2600. The recovery process may be a process of multipleprocessing layers.

According to some embodiments of the present invention, the recoveryphase may comprise re-segmenting the image, re-predicting the 3Dskeleton position, re-projecting and re-fitting the skeleton usingextended effort. According to yet further embodiments of the presentinvention, the recovery module may also decide to skip a frame or moreif the image information is corrupt.

According to some embodiments of the present invention, during recoverythe system may be adapted to detect that the object it is tracking isnot in the frame. According to yet further embodiments of the presentinvention, the system may be adapted to skip one or more frames untilthe object is back in the frame.

According to yet further embodiments of the present invention, therecovery phase may direct the system back to the initialization step.

According to some embodiments of the present invention, the recoverymodule 2600 may be adapted to perform the recovery process.

According to some embodiments of the present invention, if no error wasdetected during step 2550, the system may be adapted to update theposition of the three dimensional skeleton in accordance with theposition of the fitted skeleton, as shown in step 2700. According tosome further embodiments of the present invention, the updating processmay comprise: (1) projecting the 3D skeleton on the fitted skeleton, (2)associating the 3D skeleton with the fitted skeleton, and (3) updatingthe position of the 3D skeleton.

According to some embodiments of the present invention, the threedimensional correlation module, 1700 may be adapted to update theposition of the three dimensional skeleton.

According to some embodiments of the present invention, thethree-dimensional correlation module 1700 and the skeleton predictionmodule 1800, may use some or all of the algorithms and processes whichwere disclosed in PCT application serial number PCT/IL2005/000813, filedon 31 Jul. 2005, under the same assignee as the present application.

Turning now to FIG. 3, there is shown an exemplary HMI system inaccordance with some embodiments of the present invention, which systemmay be best described in conjunction with FIG. 4, there is shown a flowchart depicting the steps of such an HMI system.

According to some embodiments of the present invention, FIG. 3 shows a3D human machine interface (“3D HMI”), which 3D HMI may include (1) aZlens image acquisition assembly 3000, (2) an initializing module 3100,(3) an image segmentation module 3200, (4) a segmented data processingmodule 3300, (5) a fitting module 3500, (6) a scoring module 3550, (7) athree dimensional correlation module 3700, and (8) an output module3900.

According to some embodiments of the present invention, the Zlensacquisition assembly may be adapted to acquire a set of images, as seenin step 4000, wherein substantially each image is associated with adifferent point in time. According to some further embodiments of thepresent invention, the images may be of a single user or multiple users.

According to some embodiments of the present invention, the Zlensacquisition assembly may be mounted on another image acquisitionassembly, i.e. element 1000 of FIG. 1.

According to yet further embodiments of the present invention the Zlensacquisition assembly (3000) may be best described in conjunction withPCT/IL2006/001254, filed on Oct. 31, 2006, under the same assignee asthe present application and with U.S. Patent application 60/731,274 US,filed on Oct. 31, 2005, under the same assignee as the presentapplication.

According to some embodiments of the present invention, the system isfurther adapted to enter an initialization phase, as shown in step 4100,which is performed by the initialization module 3100. The initializationmodule may be adapted to detect and define the user's: (1) colors, (2)organ parameters, (3) surroundings, and (4) other parameters which areassociated with the user.

According to some embodiments of the present invention, the system maybe adapted to extract segmentation data, as shown in step 4200, whichsegmented data may comprise:

-   -   Color    -   Movement    -   Edge detection    -   Texture.

According to yet further embodiments of the present invention, the imagesegmentation module 3200 may be adapted to extract the segmented datafrom the image.

According to some embodiments of the present invention, the system maybe adapted to process the segmented data, as shown in step 4300.According to yet further embodiments of the present invention, thesegmented data may be processed in the following way:

-   -   Color—using known color parameters to detect elements and/or        light changes, for example, using skin color to detect palms and        face.    -   Movement—detecting moving elements in the frame.    -   Background removal.    -   Edge detection—detect the contours of every organ.    -   Texture—using known texture parameters to detect elements.

According to yet further embodiments of the present invention, thesegmented data processing module, 3300, may be adapted to process thesegmented data.

According to some embodiments of the present invention, the system maybe further adapted to fit portions of the extracted segmented data tothe acquired image, as shown in step 4500. According to some furtherembodiments of the present invention, the fitting process may compriseassociating portions of the extracted segmented data with dedicatedareas of the acquired image.

According to yet further embodiment of the present invention, thededicated areas may be stored in the system or may be determined duringthe initialization phase. According to yet further embodiment of thepresent invention, the dedicated areas may be specific organs of theuser (hands, head, feet) or any other element which may be acquiredduring step 3000.

According to yet further embodiment of the present invention, thefitting process may comprise testing whether the extracted segmenteddata defines parameters which are relevant to the dedicated areas.

According to some further embodiments of the present invention, thefitting module, 3500, may be adapted to the associate portions of thesegmented data with the acquired image.

According to some further embodiments of the present invention, theoutcome of this process is a “fitted image”.

According to some embodiments of the present invention, the system maybe further adapted to evaluate the quality of the fitted segmented data,as shown in step 4550. According to some embodiments of the presentinvention, evaluating the quality of the fitted segmented data maycomprise: (1) examining the processed segmented data, (2) estimating thequality of the processed segmented data, and according to the estimatedquality (3) deciding which portions of the segmented data are reliableenough to be used by the HMI system, (4) examining the fitted image, (5)evaluating the fitting quality of said image and (6) determining whetheran error has occurred during the association of segmented data.

According to some embodiments of the present invention, the scoringmodule, 3550, may be adapted to evaluate the quality of the fittedsegmented data.

According to some further embodiments of the present invention, thesystem may comprise an error detection mechanism and a recoverymechanism as was described hereinabove.

According to some embodiments of the present invention, the system maybe adapted to update the position of a three dimensional body inaccordance with the fitted image and the extrapolation of a depth mapusing the Zlens image acquisition assembly, as shown in step 4700.According to some further embodiments of the present invention, theupdating process may comprise associating the extracted depth map withthe extracted segmented data, and updating the position of thethree-dimensional body of the output model.

According to some embodiments of the present invention, the threedimensional correlation module, 3700, may be adapted to update theposition of the three dimensional body.

According to some embodiments of the present invention, thefunctionality of the three-dimensional correlation module 3700 and theZlens image acquisition assembly 3000 and particularly the extrapolationof depth from an image acquired using the Zlens apparatus may best bedescribed in conjunction with PCT/IL2006/001254, filed on Oct. 31, 2006,under the same assignee as the present application and with U.S. Patentapplication 60/731,274 US, filed on Oct. 31, 2005, under the sameassignee as the present application.

Turning now to FIG. 5, there is shown an exemplary HMI system inaccordance with some embodiments of the present invention, which systemmay be best described in conjunction with FIG. 6, where there is shown aflow chart depicting the steps of such an HMI system.

According to some embodiments of the present invention, FIG. 5 shows a3D human machine interface (“3D HMI”), which 3D HMI may include: (1) aZlens acquisition assembly 5000, (2) an initializing module 5100, (3) animage segmentation module 5200, (4) a segmented data processing module5300, (5) a scoring module 5400, (6) a projection module 5450, (7) afitting module 5500, (8) a scoring and error detection module 5550, (9)a recovery module 5600, (10) a three dimensional correlation module5700, (11) a three dimensional skeleton prediction module 5800, (12) anoutput module 5900 and an optional (13) depth extraction module 5050.

According to some embodiments of the present invention, the Zlensacquisition assembly may be adapted to acquire a set of images, as seenin step 6000, wherein substantially each image is associated with adifferent point in time. According to some further embodiments of thepresent invention, the images may be of a single user or multiple users.

According to some embodiments of the present invention, the Zlensacquisition assembly may be mounted on another image acquisitionassembly, i.e. element 1000 of FIG. 1.

According to yet further embodiments of the present invention the Zlensacquisition assembly (5000) may be best described in conjunction withPCT/IL2006/001254, filed on Oct. 31, 2006, under the same assignee asthe present application and with U.S. Patent application 60/731,274 US,filed on Oct. 31, 2005, under the same assignee as the presentapplication.

According to some embodiments of the present invention, after the systemhas acquired one or more images, the system may enter an initializationphase, step 6100, which is performed by the initialization module 5100.An initialization module may be adapted to detect and define the user's:(1) colors, (2) organ parameters, (3) surroundings, and (4) otherparameters which are associated with the user.

According to some embodiments of the present invention, the system maybe adapted to extract segmentation data, as shown in step 6200, whichsegmented data may comprise:

-   -   Color    -   Movement    -   Edge detection    -   Texture.

According to yet further embodiments of the present invention, the imagesegmentation module 5200 may be adapted to extract the segmented datafrom the image.

According to some embodiments of the present invention, the system maybe adapted to process the segmented data, as shown in step 6300.According to yet further embodiments of the present invention, thesegmented data may be processed in the following way:

-   -   Color—using known color parameters to detect elements and/or        light changes, for example, using skin color to detect palms and        face.    -   Movement—detecting moving elements in the frame.    -   Background removal.    -   Edge detection—detect the contours of every organ.    -   Texture—using known texture parameters to detect elements.

According to yet further embodiments of the present invention, thesegmented data processing module, 5300, may be adapted to process thesegmented data.

According to some embodiments of the present invention, the system maybe adapted to evaluate the quality of the segmented data, as shown instep 6400, the evaluation is performed by: (1) examining the processedsegmented data, (2) estimating the quality of the processed segmenteddata, and according to the estimated quality (3) deciding which portionsof the segmented data are reliable enough to be used by the HMI system.

According to yet further embodiments of the present invention, thescoring module, 5400, may be adapted evaluate the quality of thesegmented information.

According to further embodiments of the present invention, the systemmay be adapted to predict the position of the three dimensionalskeleton, as shown in step 6800, which position will have the best matchor correlation with the processed image. According to some furtherembodiments of the present invention the prediction may be more accuratewith the use of constraints which derive from the type of skeleton used,for example, if the skeleton is of a human figure, the head of theskeleton can't rotate 360 degrees without a motion of the shoulders.

According to some embodiments of the present invention, the predictionsequence may also use a set of dynamic and motion process.

According to some embodiments of the present invention, the threedimensional skeleton prediction module 5800 may be adapted to predictthe position of the three dimensional skeleton.

According to some embodiments of the present invention, the system maybe further adapted to extract depth using the Zlens acquisitionassembly, as shown in step 6050, the extraction of depth using a Zlensacquisition assembly is described in: (1) PCT/IL2006/001254, filed onOct. 31, 2006, under the same assignee as the present application andwith (2) U.S. Patent application 60/731,274 US, filed on Oct. 31, 2005,under the same assignee as the present application.

According to some embodiments of the present invention, the depthextraction module, 5050, may be adapted to extract depth from theacquired image.

According to some embodiments of the present invention, the system maybe further adapted to project the skeleton onto the image, as shown instep 6450. According to some further embodiments of the presentinvention, the projection may be applied in the two-dimensional plane.

According to yet further embodiments of the present invention, theprojection of the skeleton may be applied in the three-dimensional planeif module 5050 is used.

According to some embodiments of the present invention, the projectionmay be onto a three-dimensional image and/or a three dimensional cloudof points.

According to some embodiments of the present invention, the projectionmodule, 6450, may be adapted to project the skeleton onto the image.

According to some embodiments of the present invention, the system maybe further adapted to fit the segmented data with the projectedskeleton, as shown in step 6500. According to some further embodimentsof the present invention, the fitting process may comprise theassociation of portions of the segmented data with portions of theprojected skeleton.

According to some embodiments of the present invention, fitting thesegmented data may comprise associating portions of the extractedsegmented data with current skeleton parameters, which current skeletonparameters may support the associated portions of extracted segmenteddata.

According to some further embodiments of the present invention, theoutcome of this process is a “fitted skeleton”.

According to some further embodiments of the present invention, thefitting module, 5500, may be adapted to associate the segmented datawith the projected skeleton.

According to some embodiments of the present invention, the system maybe further adapted to give a score to the fitted skeleton and detecterrors, as shown in step 6550. According to some embodiments of thepresent invention, giving score and detecting errors may comprise: (1)examining the fitted skeleton, (2) evaluating the fitting quality ofsaid skeleton and (3) determining whether an error has occurred duringthe skeleton prediction process or the association of segmented data.

According to some embodiments of the present invention, the scoring anderror detection module, 5550, may be adapted to give scores and detecterrors.

According to some embodiments of the present invention, if an error wasdetected during step 6550, the system may enter a recovery phase, asshown in step 6600. The recovery process may be a process of multipleprocessing layers.

According to some embodiments of the present invention, the recoveryphase may comprise re-segmenting the image, re-predicting the 3Dskeleton position, re-projecting and re-fitting the skeleton usingextended effort. According to yet further embodiments of the presentinvention, the recovery module may also decide to skip a frame or moreif the image information is corrupt.

According to some embodiments of the present invention, during recoverythe system may be adapted to detect that the object it is tracking isnot in the frame. According to yet further embodiments of the presentinvention, the system may be adapted to skip one or more frames untilthe object is back in the frame.

According to yet further embodiments of the present invention, therecovery phase may direct the system back to the initialization step.

According to some embodiments of the present invention, the recoverymodule 5600 may be adapted to perform the recovery process.

According to some embodiments of the present invention, if no error wasdetected during step 6550, the system may be adapted to update theposition of the three dimensional skeleton in accordance with theposition of the fitted skeleton, as shown in step 6700. According tosome further embodiments of the present invention, the updating processmay comprise: (1) projecting the 3D skeleton onto the fitted skeleton,(2) associating the three dimensional skeleton with the fitted skeleton,(3) extract depth using the Zlens assembly, (4) associating thethree-dimensional skeleton with depth parameters and (5) updating theposition of the 3D skeleton.

According to some embodiments of the present invention, the threedimensional correlation module, 5700, may be adapted to update theposition of the three dimensional skeleton.

According to some embodiments of the present invention, thethree-dimensional correlation module 5700 and the skeleton predictionmodule 5800, may use some or all of the algorithms and processes whichwere disclosed in PCT application serial number PCT/IL2005/000813, filedon 31 Jul. 2005, under the same assignee as the present application.

According to some further embodiments of the present invention, thefunctionality of (1) the three-dimensional correlation module 5700 (2)the Zlens image acquisition assembly 5000 the (3) depth extractionmodule 5050 and (4) particularly the extrapolation of depth from animage acquired using the Zlens apparatus may best be described inconjunction with PCT/IL2006/001254, filed on Oct. 31, 2006, under thesame assignee as the present application and with U.S. Patentapplication 60/731,274 US, filed on Oct. 31, 2005, under the sameassignee as the present application.

According to some embodiments of the present invention, the systemsdescribed hereinabove may be adapted to receive from an exterior sourcedepth images and/or three-dimensional images. According to yet furtherembodiments of the present invention, if a depth images and/orthree-dimensional images are received the system is adapted to extractits parameters in the relevant modules.

The processes and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the desired method. The desired structure for avariety of these systems will appear from the description above. Inaddition, embodiments of the present invention are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the inventions as described herein. One of ordinary skillin the art should understand that the described invention may be usedfor all kinds of wireless or wire-line system.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents will now occur to those skilled in the art. It is,therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the invention.

What is claimed:
 1. A human machine interface comprising: an imageacquisition assembly to acquire a series of substantially consecutivesets of two-dimensional image data of a user via multiple optical paths,said image acquisition assembly comprising: a. an image sensor arrayhaving an image sensing area; and b. two or more optical paths directingoptical image information of the user onto an at least partiallyoverlapping segment of said image sensing area; a first processing unitto: (1) derive, based on the optical image information of the user fromtwo or more optical paths, estimated three dimensional coordinates ofelements of the user body, during the acquisition of at least two of thesubstantially consecutive sets of two-dimensional image data; (2)determine a movement of one or more body parts of the user between theacquisition of the at least two of the substantially consecutive sets oftwo-dimensional image data, based on a difference between the estimatedthree dimensional coordinates of the elements of the user body, duringthe acquisition of each of the at least two of the substantiallyconsecutive sets of two-dimensional image data; and a second processingunit to: (1) correlate the determined movement of one or more body partsof the user to a user input; and (2) transmit a signal representing theuser input.
 2. The human machine interface according to claim 1, whereinsaid image sensor array is a webcam.
 3. The human machine interfaceaccording to claim 1, wherein deriving, based on the optical imageinformation of the user from two or more optical paths, estimated threedimensional coordinates of elements of the user body, during theacquisition of at least two of the substantially consecutive sets oftwo-dimensional image data, includes comparing appearances of theelements of the user body from the different optical paths.
 4. The humanmachine interface according to claim 3, wherein comparing appearances ofthe elements of the user body from the different optical paths includestriangulation.
 5. The human machine interface according to claim 1,wherein each of said two or more optical paths comprises a distinctoptical element.
 6. The human machine interface according to claim 5,wherein said distinct element is a distinct filter.
 7. The human machineinterface according to claim 5, wherein said distinct element is adistinct grid.
 8. The human machine interface according to claim 1,wherein said optical paths direct optical image information of the useronto a common segment of said image sensing area.
 9. A method of humanmachine interfacing, comprising: acquiring, by an image acquisitionassembly, a series of at least two substantially consecutive sets oftwo-dimensional image data of a user, via multiple optical paths,wherein said image acquisition assembly comprises: an image sensor arrayhaving an image sensing area; and two or more optical paths directingoptical image information of the user onto an at least partiallyoverlapping segment of said image sensing area; deriving, by processingcircuitry and based on the optical image information of the user fromtwo or more optical paths, estimated three dimensional coordinates ofelements of the user body, during said acquisition of at least two ofthe substantially consecutive sets of two-dimensional image data;determining, by the processing circuitry and based on a differencebetween the estimated three dimensional coordinates of the elements ofthe user body, during the acquisition of each of the at least two of thesubstantially consecutive sets of two-dimensional image data, a movementof one or more body parts of the user, between the acquisition of the atleast two of the substantially consecutive sets of two-dimensional imagedata; correlating, by the processing circuitry, the determined movementof one or more body parts of the user to a user input; and transmittinga signal representing the user input.
 10. The method according to claim9, wherein said image sensor array is a webcam.
 11. The method accordingto claim 9, wherein deriving, based on the optical image information ofthe user from two or more optical paths, estimated three dimensionalcoordinates of elements of the user body, during the acquisition of atleast two of the substantially consecutive sets of two-dimensional imagedata, includes comparing appearances of the elements of the user bodyfrom the different optical paths.
 12. The method according to claim 11,wherein comparing appearances of the elements of the user body from thedifferent optical paths includes triangulation.
 13. The method accordingto claim 9, wherein each of said two or more optical paths comprises adistinct optical element.
 14. The method according to claim 13, whereinsaid distinct element is a distinct filter.
 15. The method according toclaim 13, wherein said distinct element is a distinct grid.
 16. Themethod according to claim 13, wherein said optical paths direct opticalimage information of the user onto a common segment of said imagesensing area.